There has been a growth in the use of hyperspectral cameras for aerial imaging, environmental monitoring, forensic science, forestry, agriculture as well as in military and industrial applications. Hyperspectral cameras normally also cover a wavelength range outside the visible thus making the design of the optical system very challenging. There are different principles of hyperspectral imagers, but in this patent application a “hyperspectral camera” is defined as a hyperspectral imager of the “push broom” type.
One of the most important limiting factors in a hyperspectral imaging system is spatial misregistration as function of wavelength. In hyperspectral data, each spatial pixel is supposed to contain the signals for different spectral bands captured from the same spatial area. This type of misregistration occurs when either the position of a depicted area (corresponding to one pixel) changes as function of wavelength (this error is commonly referred to as “keystone”), or the borders or objects in the depicted area are blurred differently for different spectral channels, which is caused by variation in the point spread function (PSF) as function of wavelength. In practice this means that information from different positions on the scene depicted can be intermixed and resulting in an erroneous spectrum for spatial image pixels positioned close together.
Another effect or limiting factor is a similar spectral misregistration commonly referred to as “smile”. This effect will partly shift the spectrum of a depicted point.
Most push-broom spectrometers operate as follows: they project a one-dimensional image of a very narrow scene area onto a two-dimensional pixel array. A dispersing element (a diffraction grating or a prism) disperses light in such a way that, instead of a one-dimensional image elongated in X direction, each column of the array contains spectrum of one small scene area as shown in FIG. 2. The two-dimensional image of the scene is then obtained by scanning. The final data form a so-called “datacube”, which represents a two-dimensional image with the third dimension containing the spectral information for each spatial pixel.
Due to various optical aberrations in real-world hyperspectral cameras, the spectral information is not captured perfectly. Spectral misregistration occurs when a given pixel on the sensor is expected to receive light from a certain spectral band but instead receives slightly wrong wavelengths. Spatial misregistration occurs when a pixel on the sensor receives a certain spectral band but looks at slightly wrong area of the scene. Misregistration is discussed in more detail in the article “Spectral and spatial uniformity in pushbroom imaging spectrometers” by Pantazis Mouroulis, 1999.
Both types of misregistration may severely distort the spectral information captured by the camera. Therefore they should be corrected to a small fraction of a pixel. In hyperspectral cameras for the visible and near-infrared region (VNIR) spectral misregistration can be corrected by oversampling the spectral data and resampling it in postprocessing (since most of the sensors have many extra pixels in the spectral direction, such a camera will still have good spectral resolution). However, it is normally desirable to correct the spatial misregistration in hardware as well as possible.
Spatial misregistration in hyperspectral cameras is caused by two factors:
1). Variation in position of the PSF's centre of gravity as a function of wavelength, usually called spectral keystone. This is shown in FIG. 4.
2). Variation in size and shape of PSF as a function of the wavelength. This is shown in FIG. 5.
Since the positions of PSF's centre of gravity for different wavelengths must not differ by more than a small fraction of pixel size (even as small deviation as 0.1 of a pixel may introduce noticeable errors in the measured spectrum), optical design of such systems is very challenging. Keeping the size and the shape of PSF similar for all wavelengths is perhaps even more difficult.
In general, pixel count in hyperspectral cameras is very modest (compared to traditional imaging systems such as photocameras), since it is not enough just to produce a reasonably sharp image, and optical aberrations have to be corrected at subpixel level. Optical design becomes extremely difficult if higher spatial resolution is required.
Another serious challenge is to build the designed camera. Manufacturing and centration tolerances for hardware corrected cameras (even with relatively modest spatial resolution of 300-600 pixels) can be very tight.
As a result of such strict requirements to the image quality, the new hyperspectral cameras more or less converged to a few designs which offer somewhat acceptable correction of spatial misregistration.
Offner relay with a convex diffraction grating has magnification −1 and is often used in hyperspectral cameras (“Optical design of a compact imaging spectrometer for planetary mineralogy” by Pantazis Mouroulis et al, Optical Engineering 466, 063001 June 2007; U.S. Pat. No. 5,880,834). It can be designed to have reasonably small spectral keystone. Variations in PSF's size and shape can be corrected to some extent, but there are too few optical surfaces to make this kind of correction perfect. In a real system manufacturing and centration tolerances degrade the spatial misregistration further. Even though Offner relay is not very sensitive to decentration, the tolerances can be very tight in cameras with high spatial resolution. The minimum i.e., the fastest F-number in Offner cameras is limited to approx. F2.5. Polarisation dependency of the diffraction grating may be a problem.
Dyson relay is another example of an optical system used in hyperspectral imaging. (Optical design of a coastal ocean imaging spectrometer, by Pantazis Mouroulis et al, 9 Jun. 2008/Vol. 16, No. 12/OPTICS EXPRESS 9096). It has magnification −1 and uses a concave reflective grating as a dispersive element. There is also at least one refractive element. The F-number can be significantly lower (faster) than in Offner systems. The system can be quite small. However, extremely tight centration requirements make it difficult to achieve low misregistration errors even with low resolution sensors. Both the slit and the detector face the same optical surface, therefore stray light is often a problem. Also, it is a challenge even to place the detector close enough to the first optical surface (as close as it is required in the Dyson design). Due to extremely tight tolerances and practical difficulties in placing the detector, the resolution of Dyson cameras is not particularly high.
Design and manufacturing of good foreoptics (i.e. the foreoptics which would take full advantage of low misregistration error in the following relay) for both Offner and especially Dyson relays is very challenging.
Some manufacturers base their hyperspectral cameras on various proprietary lens systems. Performance of such cameras is more or less similar to the performance of the Offner based cameras.
In a push-broom hyperspectral camera rays with different wavelengths are focused on different parts of the sensor. Therefore, compared to more traditional imaging systems (such as photolenses), sensor tilt can be introduced (U.S. Pat. No. 6,552,788 B1). This tilt can be used as an additional parameter when correcting keystone, smile, and PSF variations in the optical system. Of course, the tilt will not eliminate keystone etc. completely—it merely offers more flexibility in optimizing system's performance. However, in relatively complex systems, where there are many such parameters available already, introduction of this additional parameter may not lead to any significant improvements in keystone, smile, and PSF variation correction.
A known alternative to precise correction of spatial misregistration in hardware is resampling. Since the most challenging requirements are lifted in resampling cameras, the optical design becomes similar to traditional imaging optics. This gives a possibility to design optics with lower (faster) F-number, high resolution etc. The negative side is the fact that the residual spatial misregistration after resampling with approx. 1× factor is quite large. In order to bring it down to acceptable level downsampling by factor 2× or more is usually required. Therefore the full sensor resolution is not utilized. Also, the necessity to capture two times more pixels than in case of hardware corrected systems, may slow down the frame rate and processing speed.
Variable filter camera is another way to capture hyperspectral data. The principle of operation is described in the U.S. Pat. No. 5,790,188. This camera can be quite small compared to the other hyperspectral cameras. However, it may be necessary to resample the captured data, unless the scanning motion is stable with subpixel accuracy. Also, the spectral resolution becomes very limited if a reasonably low F-number is required.
If a hyperspectral camera has to work in a very wide spectral range, it may be beneficial to split this range between two or more sensors, as described in the NASA PIDDP Final Report (May 1, 1996), A Visible-Infrared Imaging Spectrometer for Planetary Missions. Example of such an imaging spectrometer is shown on FIG. 3 of this report. Since both sensors share foreoptics and slit, it is possible to get very good spatial coregistration in the along-the-track direction. However, in the across-the-track direction (where keystone is normally measured) it will be nearly impossible to achieve good (i.e. a few percent of a pixel) coregistration. This drawback of the multiple sensor approach is explained in the Final Report of the Concept for Future Visible and Infrared Imager>>Study, by Astrium GmbH for ESA, Doc. No FI-RP-ASG-0007 (Chapter 4.5.3, Page 4-33).
If the slit 2040 in the camera from FIG. 2 is replaced by a thin plate with pinholes arranged in a two-dimensional array, it becomes possible to capture the data cube in one exposure without scanning. This approach together with possible enhancements is thoroughly described in the US Patent Application Publication Nos. 2006/0072109 A1 and 2008/0088840 A1. While being great for some applications (such as multispectral video), this approach severely limits spatial and/or spectral resolution of cameras (if compared with push-broom approach). Also, cameras built on this principle are just as prone to smile and keystone errors as the previously described push-broom cameras.
Necessity to correct spectral keystone to a very small fraction of a pixel is the principal driver restricting the spatial resolution of the existing hyperspectral cameras, possible design approaches, speed (i.e. light gathering ability), and the final data quality. Resampling hyperspectral cameras, while being free of the first three restricting factors, normally require downsampling of the captured data by a large factor, which reduces the resolution (i.e. pixel count) of the final data. In other words, even though it is possible to design and manufacture a resampling camera with very sharp optics and very high pixel count, the captured data have to be downsampled by a relatively large factor in order to match the data quality from a hardware corrected camera. This downsampled data may still have higher pixel count than the data from a hardware corrected camera, but clearly the full resolution of the sensor will not be utilized.
To sum up this means that in state-of-the-art systems the requirement for keystone correction in the optics typically limit the maximum resolution, the light collecting capabilities of the system, not to mention the number of feasible optical design solutions.
Even when the keystone has been corrected as well as possible using all available techniques, the remaining keystone will still be significant for a vast majority of systems. In the few systems where the keystone appears to be closer to acceptable the overall resolution or pixel count is relatively low. The cause of this is that the precision of the keystone correction is linked to the pixel count for current state-of-the-art systems. Physically in any given system keystone is a certain fraction of the image size. When one increases the pixel count for a given image size, keystone as a fraction of the pixel size will therefore increase.